Open analytical ecosystems for digital humanities

Open science principles

• Emphasis on research process

• Transparency (data, methods, reporting)

• Reproducibility

• Openness (unlimited access and reuse)

• New modes of collaboration and initiatives

• Access to data is an institutional question. Using and tidying up the data is a research question

• Automation vs. point-n-click ?

Library catalogues: the data

https://github.com/rOpenGov/estc

• English Short Title Collection (ESTC) from British Library
• 50161 documents on history (~10% of the ESTC)
• Finland (Fennica), Europe (other catalogues), etc. etc.
• New use for old databases
• Quantitative knowledge production: book sizes, publishers, authors..

Publishing “history” in Britain and North America 1470-1800

Research questions

1. Who wrote history?
   
2. Where was it published?
3. How does the publishing of history change over the early modern period?

ESTC raw data

Hierarchical information, only some fields relevant for our study

Workflow, step by step

• Retrieve -> Parse -> Tidy up
• Enrich & Analyse
• Report & Use

Load the data and tools

Load the data and tools in R:

load("df.RData")
library(bibliographica)
kable(t(df.orig[22495, ]))

Polishing page counts

Raw page counts

rawpages <- as.character(unique(df.orig[sample(nrow(df.orig), 6), "300.a-Extent"]))
kable(rawpages)

Polish page counts

polish_pages(rawpages)$total.pages

## [1]  12 220   8   2   2   8

Document dimension field

kable(as.character(sample(unique(df.orig[, "300.c-Dimensions"]), 6)))

Polish document dimensions

Pick dimension information

kable(polish_dimensions("10 cm (12⁰)"))

Fill missing dimensions

Estimate missing dimensions

kable(polish_dimensions("10 cm (12⁰)", fill = TRUE))

Publication place

Many versions of London:

x <- as.character(df.orig[, "260.a-Place of publication"])
top_plot(x[grep("London", x)], ntop = 20)

In total 374 unique places with the string London - tidying up and synonyme lists !

Ambiguous authors

Author gender

Enriching data by external information

as.matrix(get_gender(polish_author(sample(unique(df$author.name), 20))$first)$gender)

##           [,1]    
## samuel    "male"  
## richard   "male"  
## william   "male"  
## gamaliel  "male"  
## robert    "male"  
## charles   "male"  
## john      "male"  
## prudencio "male"  
## thomas    "male"  
## hyde      "male"  
## thomas    "male"  
## mary      "female"
## thomas    "male"  
## zachary   "male"  
## henry     "male"  
## thomas    "male"  
## william   "male"  
## robert    "male"  
## n         NA      
## charles   "male"

Workflow

Who wrote history?

• Authors (number of titles / paper use / life years)

• Times 1470 - 1800 ?

• Places: London, Ireland, Scotland, North America.. ?

• Language ?

• Gender ?

Who wrote history?

Top-10 authors (number of titles)

top_plot(df, "author.unique", 20)

Who wrote history?

Top-10 female authors (number of titles)

Who wrote history?

Title count vs. paper consumption

Document count vs. paper for top authors

ggplot(df2, aes(x = docs, y = paper)) + geom_text(aes(label = author.unique), size = 4)

Who wrote history?

Gender distribution for authors over time. Note that the name-gender mappings change over time. This has not been taken into account yet.

## 
## female   male 
##  0.026  0.974

Who wrote history?

Other questions to explore

• Publishing times 1470 - 1800 ?

• Places: London, Ireland, Scotland, North America.. ?

• Language ?

df2 <- df %>% filter(publication.place == "London")
df2 <- df %>% filter(language == "French")
df2 <- df %>% filter(publication.year >= 1700 & publication.year < 1800)
top_plot(df2, "author.unique", 10)

2. Where was history published ?

Top-10 places (number of titles)

top_plot(df, "publication.place", 10)

Where was history published ?

df2 <- df %>% filter(publication.country %in% c("France", "Germany")) %>%
    group_by(publication.decade, publication.country) %>%
    summarize(paper = sum(paper.consumption.km2, na.rm = TRUE), docs = n()) 
p <- ggplot(df2, aes(x = publication.decade, y = docs, color = publication.country)) +
     geom_point() + geom_smooth()
print(p)     

Where was history published ?

Title count vs. paper

ggplot(df2,
     aes(x = log10(1 + docs), y = log10(1 + paper))) +
     geom_text(aes(label = publication.place), size = 3) +
     scale_x_log10() + scale_y_log10() 

Where was history published ?

Scotland, Ireland, US comparison:

df2 <- df %>%
    filter(!is.na(publication.country)) %>%
    group_by(publication.country) %>%
    summarize(paper = sum(paper.consumption.km2, na.rm = TRUE),
          docs = n()) %>%
    arrange(desc(docs)) %>%
    filter(publication.country %in% c("Scotland", "Ireland", "USA"))

Where was history published ?

p1 <- ggplot(subset(melt(df2), variable == "paper"), aes(y = value, x = publication.country)) + geom_bar(stat = "identity") + ylab("Paper consumption")
p2 <- ggplot(subset(melt(df2), variable == "docs"), aes(y = value, x = publication.country)) + geom_bar(stat = "identity") + ylab("Title count")
grid.arrange(p1, p2, nrow = 1)

3. How does the history publishing change in the early modern period ?

What can we say about the nature of the documents? Pamphlets (<32 pages) vs. Books (>120 pages) ? Book size statistics and development over time

Nature of the documents

Nature of the documents

Estimated paper consumption by document size

Nature of the documents

Document sizes over time

Serious statistical analysis (also in the Humanities)

• ~80 % of statistical analysis is tidying up of the data. Too often neglected and implicitly assumed by many tools. We provide new efficient tools also for this

• With open data principles, no need to reinvent the wheel for the same (or similar) datasets

• Things become stable. The research tool is corrected and perfected when it is transparent & potentially used also by others

• Possibilities of reuse with similar datasets is great

• Automatization allows reporting with minimal human intervention

Open science in (digital?) humanities

• Innovative use of computational and statistical methods

• New tools for old questions derived from the discipline itself

• Vast amounts of useful data not being shared or utilized

• Open access not enough. We need open sharing of research data and methods to study “traditional” questions

These slides are automatically generated as well

Barriers to open science in the humanities

• Institutions that hold the raw data are reluctant to give full access to data (even to researchers of the same institution). Why?

• Research process is not opened and research data is not shared in the Humanities. Transparency, reproduction, collaboration, new initiatives are missing. Why?

• Short answer: Cultural change takes time. We need concrete examples in the core field of the Humanities that actually prove OPEN DATA PRINCIPLES as useful.

• Statistical software for computational social sciences & humanities
• Open source / Fully transparent
• Reproducible workflows
• Preprocessing, enrichment, integration, analysis, visualization, reporting..
• Strong developer community (Finland + International)
• Practical & low-cost research tools rather than polished software product
• Based on analogous ecosystems from other fields (bioinformatics, ecology..)

Thomason tracts 1640-1660

Gatherings and page counts

Page counts

Page count: distribution for documents with different sizes.

How does the history publishing change in the early modern period ?

Nature of the documents

Estimated title count by document size

Nature of the documents

Top authors

Nature of the documents

Top authors title count

How does the history publishing change in the early modern period ?

• Map visualization ?

How does the history publishing change in the early modern period ?

Top-4 places (title count), mean page count over time.